Risk of Rabies and Implications for Postexposure Prophylaxis Administration in the US

This decision analytical model evaluates the risk of rabies and implications for postexposure prophylaxis administration in the US.

Bat species. Forty-two percent (N = 106,501) of 253,028 bat samples in the data had information about the species. We removed vampire bats and mega bats which are not currently found in the wild in the United States (N = 55, representing 4 species). We also removed 17 species with high uncertainty in positivity rates due to small sample sizes (95% exact binomial confidence intervals of positivity rates greater than or equal to 0.25), resulting in 106,346 samples across 28 species.
Positivity rates. Positivity rates will be higher than the RABV infection prevalence in the general animal population as the samples are submitted via passive surveillance. For bats, cats, and dogs, we calculated jurisdiction-specific positivity rates because sufficient data were available across all jurisdictions (eFigure 3). We also calculated jurisdiction-specific rates for raccoons, skunks, foxes, arctic foxes (Alaska only), and the small Indian mongoose (Puerto Rico only) because distinct RABV variants are associated with these reservoir species (eFigure 4). For jurisdictions with high uncertainty of positivity rates for certain animals (the width of the 95% exact binomial confidence intervals was greater than or equal to 0.25), we used regional, rather than jurisdiction-specific, estimates. The regions were defined according to the primary rabies reservoir in each jurisdiction (skunk, bat, or raccoon). 4 We calculated two regional positivity rates each for cattle, equine, sheep and goats, rodents of unusual size, and hoofed animals, pooling samples across jurisdictions according to whether terrestrial RABV variants are present (eFigure 5). Rabies risk is considered higher for these species in states where terrestrial RABV variants are found. We classified Washington, Oregon, Nevada, Idaho, Illinois, Indiana, Mississippi, and Utah as being free from terrestrial RABV. 4 For the remaining animal categories, we pooled samples across all jurisdictions to calculate national estimates of the positivity rates (eTable 1). We also calculated positivity rates for bat species (eFigure 6).
Parameter estimates for health status. New Hampshire and Arkansas were the only two jurisdictions that provided data regarding an animal's health status. New Hampshire had 489 and 506 samples with a conclusive (positive or negative) test result for 2018 and 2020, respectively. For those years, presence of rabies symptoms was reported as a binary variable (yes/no). A list of symptoms (what constituted a "yes") was not provided with the data. For 2018, only 94 samples had non-missing symptom information, and for 2020, 84 samples had non-missing symptom information, for a total of 178 samples. Eight out of 178 total samples had reported symptoms.
In 2020, Arkansas had 676 samples with a positive or negative test result. Of those, symptom information was reported (non-missing) for 238 animals as free text. We removed 44 samples in which the circumstances of the exposure were reported rather than symptoms (e.g., hit by car or found dead). For the remaining 194 samples, we separated animals that were apparently healthy from those that were ill or acting strangely by searching the free text using the following key words: "none," "no symptoms," and "healthy." Samples that were classified as ill or acting strangely had words including "out during day" (for nocturnal species), "slobbering," "staggering", "paralysis", and "unusual viciousness," among others.
We included USDA data for animal health status because they contain a larger proportion of healthy animals compared to data from the jurisdictions due to active surveillance activities, which include trapping and euthanasia regardless of human or other animal exposure (i.e., to pets). From 2015 -2020, 41,716 samples with a positive or negative test result were reported by USDA. Of those, 2,750 had non-missing information for the variable "why euthanized"; 412 animals were euthanized for "abnormal behavior," while 2,338 exhibited "normal behavior." We combined the data from New Hampshire, Arkansas, and USDA and constructed a 2x2 table of health status by RABV test result.
Parameter estimates for exposure circumstances. Similarly, Arizona was the only jurisdiction that reported data on the circumstances of the exposure for its RABV samples. Arizona reported information about the circumstance of exposure for RABV samples submitted from 2018 -2020. A positive or negative test result was reported for 618, 562, and 440 samples in 2018, 2019, and 2020, respectively. We restricted the data to only include instances where "yes" was selected for the variable "was there possible human exposure" and the number of humans with possible exposure was at least 1. We then excluded samples for which the variable "was the contact bite provoked" was missing, resulting in 188 samples for 2018, 186 for 2019, and 122 for 2020. Of the 496 total samples, 358 exposures were provoked, while 138 were unprovoked. We constructed a 2x2 table of exposure circumstance by RABV test result.
Literature review. We began our review by searching Google Scholar for the terms "rabies" AND "death" AND "probability" on October 19, 2021. We checked the references of relevant publications to identify as many original data sources as possible. We limited our search mainly to English-language publications except for one key reference, which was written in French. Articles were excluded if they did not report the probability of death from rabies by anatomical location and severity of exposure. We updated our search on April 11, 2023 using the following additional search terms: "virus" and "animal." We also expanded the search to PubMed but did not find any additional publications.
We expected to find few, if any, publications from the United States as there have only been 1 -2 human rabies deaths per year since 1960. 11 Survey. Survey data were collected online through Google Forms. A copy of the survey is included in Part 3.
Respondents were asked whether they agreed with the following statement about a possible RABV exposure in their jurisdiction: post-exposure prophylaxis (PEP) is recommended. They could select from four options: strongly agree, agree, disagree, and strongly disagree. Only the following information was provided about each scenario: • Animal species/common name • Whether the exposure was provoked • Whether the animal is healthy • Whether the animal is up to date on its rabies vaccines (domestic animals only) Survey scenarios were selected to represent a range of risk probabilities; however, scenarios with lower probabilities were overrepresented to estimate a risk threshold (eFigure 7). Seven questions asked about a scenario with a probability corresponding to a pooled, national estimate. These included: small rodents, lagomorphs, and Eulipotyphla; ferret; armadillo and opossum; non-native wild; mesocarnivores; bobcat; and swine. We did not present the results of the final question about the urgency with which PEP should be administered given that an exposure had occurred and PEP had been recommended, as some respondents found it confusing.
We first piloted the survey among CDC's Rabies Epidemiology Team (N = 9). Seven team members conduct >1,500 rabies consultations each year through a public-facing hotline and email box. Each member took the survey twice, first representing Georgia and second representing another jurisdiction. The average completion time of the survey was 12 minutes in the pilot study. We performed a preliminary analysis of the pilot survey data to inform the larger survey (results not shown).
For the real survey, the target sample size was at least two respondents from each state, except for Hawaii, plus Washington D.C. and Puerto Rico (N = 102). We wanted to get good geographic coverage and recognize that only 1 -2 individuals work on rabies at the state health department level per jurisdiction. Ideally, these individuals would have at least five years of experience making professional recommendations regarding rabies PEP. The NASPHV Executive Board emailed the survey link to state public health partners, and data collection was open from May 24, 2022 -August 5, 2022.
Cleaning survey data. We had 1,200 observations (24 scenarios * 50 respondents) after matching the scenarios with the quantitative estimates; however, 6 observations were removed from Puerto Rico involving skunks, raccoons, and foxes because no samples were available for those animals in the surveillance data from 2011 -2020. Alternatively, we could have assumed Puerto Rico had the same risk for these scenarios as Florida due to similar climate and geographic proximity. See eResults for details about the survey respondents.
Accounting for uncertainty for animals with no reported positive cases. To capture uncertainty in the data for animal categories with positivity rates of 0, such as pocket pets, we used the fact that a beta distribution is a conjugate prior distribution for the binomial distribution. 12 A prior distribution and likelihood function are considered conjugate if by multiplying them together and rescaling, a posterior distribution in the same family as the prior distribution is obtained. We combined the binomial likelihood with a beta(1,1), or uniform prior distribution, to obtain a beta posterior distribution and then sampled from the posterior distribution. 12 Determining the decision boundary for p in logistic regression. Choosing p, the threshold for the decision in logistic regression, depends on the prediction context and is therefore subjective. In this analysis, p represents whether PEP would be recommended for different Pr(rabid|exposure) based on subject matter experts' opinion from the survey. We performed 10-fold cross-validation and selected the decision boundary with the highest F-score. F-score is the harmonic mean of precision and recall (sensitivity) and is considered the best performance metric to use with unbalanced data. 13 We randomly split the survey data into 10 equally sized folds. For each fold, we trained the logistic regression model on the remaining 9 folds (as in the main text, with the natural log of Pr(rabid|exposure) as the predictor and PEP yes vs no as the outcome) and saved the model coefficients. Then we used the coefficients to make predictions on the fold that was not used to train the model (the testing set). We saved the number of false positives (FP), false negatives (FN), and true positives (TP) corresponding to decision boundaries ranging from 0.35 to 0.75. We summed the FPs, FNs, and TPs across the 10 folds and calculated the F-score as (2 * TP) / (2 * TP + FP + FN) for each decision boundary. This approach to calculating F-score is preferred over calculating the mean of F-score across the folds because it avoids the problem of undefined precision or recall. 13 Sensitivity analyses on survey data. We repeated the analysis of the survey data by only including respondents with at least 9 years of professional experience recommending PEP (N = 22 experts and 528 observations).
As survey responses consisting of "agree" and "disagree" may be more subjective than "strongly agree" and "strongly disagree", we also repeated the logistic regression analysis of the survey data using only "strongly agree" (N = 499) and "strongly disagree" (N = 61) responses.
We investigated survey scenarios in which respondents selected "disagree" or "strongly disagree" for PEP, but the Pr(rabid|exposure) was above our estimated threshold. These scenarios represent false positives in which our analysis suggests PEP is needed, but experts would not recommend it in practice.
Shiny apps. The first Shiny app we developed applies to all animal exposures in the United States, while the second applies only to bat species. We provided the risk threshold in the apps to interpret the estimates for Pr(rabid|exposure). For Pr(death|exposure), users can compare the risks for possible RABV exposures to the probability of dying within one year as estimated by the U.S. Social Security Administration by age and sex. 14 We used the 2019 period life table, which was used in the 2022 Trustees Report. We also compared Pr(death|exposure) to the lifetime probabilities of dying from other causes. The National Safety Council publishes the lifetime odds of death for selected causes in the United States. 15 We used the most current estimates, which were calculated using the population and life expectancy for the year 2020, and converted the odds to probabilities.
The two Shiny apps work by filtering down a dataset based on the user's selections. Options include U.S. jurisdiction, animal category, provoked vs unprovoked exposure, animal being apparently healthy vs ill or acting strangely, and animal's vaccination status. The bat species app is agnostic to geographic location and was requested by CDC partners at the National Park Service. Even if a bat is unavailable, sometimes park rangers can guess the species based on the specific geographic location of the exposure in the national parks. The bat species tool was also created because most human rabies deaths in the United States are attributed to bat exposures, and certain species are known to be high-risk for rabies. 16 Number needed to treat (NNT). Assuming PEP reduces the risk of rabies death by 100%, 17 NNT = 1/Pr(death|exposure). We calculated NNT using the median estimates of Pr(death|exposure). To obtain the PEP cost per death averted, we multiplied NNT by $3,800, the estimated average cost of PEP. 18

eFigure 1. Flow Chart Showing Exclusion Criteria and Number of Rabies Virus (RABV) Samples Included in the Analysis, 2011 to 2020, US eTable 1. Animal Categories for Rabies Risk Model
Animals were classified into 23 mutually-exclusive risk groups for rabies based on existing guidance, taxonomy, diet, and geographical location (native vs non-native wild). 935,881 total samples submitted for rabies virus testing were available for analysis from 2011 -2020.

Category
Animal

. Temporal Trends in Positivity Rates of Rabies Virus Across Animal Categories, 2011 to 2020
Positivity rates were calculated from passive surveillance data as the number of samples that tested positive divided by the total number of samples with a positive or negative test result. The total number of samples is shown in each plot. See eTable 1 for the list of species that make up these categories.

eFigure 3. Positivity Rates of Rabies Virus for Bats, Cats, and Dogs by US Jurisdiction, 2011 to 2020
Color indicates main rabies virus reservoir in each jurisdiction. Positivity rates were calculated from passive surveillance data as the number of samples that tested positive divided by the total number of samples with a positive or negative test result. Median and 95% confidence interval (binomial exact test) are shown.

eFigure 4. Positivity Rates of Rabies Virus for Raccoons, Skunks, and Foxes by US Jurisdiction, 2011 to 2020
Color indicates main rabies virus reservoir in each jurisdiction. A regional estimate of the positivity rate of skunks was used for five jurisdictions, and for foxes, a regional estimate was used for 10 jurisdictions; for these jurisdictions, the original estimate is shown by transparent points and error bars, while the regional estimate is shown by solid points and error bars. Positivity rates were calculated from passive surveillance data as the number of samples that tested positive divided by the total number of samples with a positive or negative test result. Median and 95% confidence interval (binomial exact test) are shown. From 2011 -2020, no samples of raccoons, skunks, or foxes were available for Puerto Rico, while no samples of raccoons or foxes were available for Alaska.

eFigure 5. Positivity Rates of Rabies Virus (RABV) for Cattle, Equine, Hoofed Animals, Rodents of Unusual Size (ROUS), and Sheep and Goats, 2011 to 2020
Positivity rates were calculated by pooling samples across all jurisdictions ("National") as well as across jurisdictions according to the presence of terrestrial RABV variants. Positivity rates were calculated from passive surveillance data as the number of samples that tested positive divided by the total number of samples with a positive or negative test result. Median and 95% confidence interval (binomial exact test) are shown.

eFigure 6. Positivity Rates of Rabies Virus by Bat Species, 2011 to 2020
Positivity rates were calculated as the number of samples that tested positive divided by the total number of samples with a positive or negative test result. Median and 95% confidence interval (binomial exact test) are shown. Positivity rates for all bats as well as those with unknown species are also shown.

eFigure 7. Pr(rabid|exposure) Associated With Survey Scenarios
Median and 95% credible intervals are shown for each scenario, which is represented on the x-axis. Scenarios were ordered by increasing Pr(rabid|exposure). As an example, the point on the far left of the plot represents the least risky scenario: an exposure to cat in Illinois that was apparently healthy, vaccinated, and provoked. Colors indicate whether the estimate was jurisdiction-specific or national.

Rabies Postexposure Prophylaxis Recommendation Questionnaire
For each scenario, please indicate whether you agree with the following statement: post-exposure prophylaxis (PEP) is recommended. Assume that the animal is not available for testing or observation, that an exposure has occurred, and that the exposure occurred in your state. Scenarios are presented in the following way: Animal species / whether the exposure was provoked / whether the animal is healthy / vaccination status of the animal (domestic animals only).
The final question asks about the urgency with which PEP should be administered given that an exposure has occurred and PEP has been recommended. Name: Email: State: At which level of public health do you work?
Which option best describes your education or experience (select all that apply)?
Approximately how many years of experience making professional recommendations regarding rabies PEP do you have?

eAppendix 3. Rabies Risk Assessment Case Studies
State and local public health officials often consult CDC's rabies epidemiology team on rabies exposures in their jurisdictions via phone or email. We collected examples of consults that occurred in 2022 and compared the model recommendation in each case with the actual decision that was made. Examples were also submitted by colleagues on the National Association of Public Health Veterinarian's (NASPHV) Executive Board.
Note that given the fatal nature of rabies, public health and clinical decisions are generally made based on an individual's probability of exposure, Pr(rabid|exposure), rather than their probability of dying from an exposure, Pr(death|exposure). Nevertheless, we provided both estimates for the scenarios in this supplement whenever possible.
Case study 1: New York sloth Scenario. A child reached into a sloth's enclosure in New York and pulled its tail [provoked], prompting the sloth to scratch the child with its teeth. The sloth was apparently healthy but unvaccinated. The child was unable to be reached for post-exposure prophylaxis (PEP), and health officials needed to decide whether to kill and test the animal, which had been at the facility for four months (NASPHV recommends that wild-caught animals with public contact be quarantined for a minimum of six months). 27 Estimates. Although no sloths had been tested for rabies virus (RABV) in the United States from 2011 -2020, the animal would be classified as non-native wild, which is a pooled, national estimate. The Pr(rabid|exposure) was 0.0013 (95% CrI: 0.0005 -0.0023), and Pr(death|exposure) was 0.00007 (95% CrI: 0.00003 -0.00011) for the child, assuming the exposure was to the upper extremities. Tool recommendation. Considering the uncertainty in the estimate, Pr(rabid|exposure) overlapped with the risk threshold of 0.0004 (95% CI: 0.0002 -0.0006). Assuming the child was a 10-year-old male, the probability of dying within one year from any cause is estimated to be 0.000097, 14 which is higher than the median Pr(death|exposure) from the animal exposure. If the child could be located, PEP could be considered based on Pr(rabid|exposure); PEP would be recommended based on Pr(death|exposure). Outcome. Health officials elected for a 30-day quarantine of the sloth.
Case study 2: Pennsylvania raccoon Scenario. A woman in Pennsylvania found her grandchildren outside trying to "save" a raccoon that was showing neurological signs, including excessive salivation. The children's efforts included trying to give the raccoon water [provoked]. The raccoon was killed, and testing was inconclusive. Estimates. The Pr(rabid|exposure) was 0.25 (95% CrI: 0.23 -0.27). Not enough information was available to determine Pr(death|exposure). Tool recommendation. If the grandchildren were exposed to the raccoon (the animal's saliva contact broken skin or mucous membranes), PEP would be recommended by the tool as the Pr(rabid|exposure) is 625 times the risk threshold. Outcome. PEP was recommended.

Case study 3: Pennsylvania dog
Scenario. A dog presented to a veterinary clinic in Pennsylvania with acute pain (thought to be due to a kidney infection) and bit a veterinary technician on the hand [likely provoked if being handled]. The dog was not up to date on its rabies vaccinations, and the owner refused to provide information about how the dog was housed. Estimates. The Pr(rabid|exposure) was 0.0054 (95% CrI: 0.0034 -0.0077), and the Pr(death|exposure) was 0.0013 (95% CrI: 0.0007 -0.0022). Tool recommendation. The Pr(rabid|exposure) is above the risk threshold of 0.0004 (95% CI: 0.0002 -0.0006). Assuming the veterinary technician was a 30-year-old female, the probability of dying within one year from any cause is estimated to be 0.00081, 14 which is lower than the median Pr(death|exposure) from the animal exposure. Thus, based on both Pr(rabid|exposure) and Pr(death|exposure), PEP would be recommended. Outcome. The dog died one day after the bite. The owner initially refused to submit the dog for testing, but eventually relented. There was a delay of about five days after the bite before testing results were anticipated. The recommendation was to wait for testing results, but the victim decided to start PEP out of an abundance of caution. The dog tested negative for RABV.

eFigure 8. Risk Thresholds (Vertical Colored/Dashed Lines) Associated With Each Value of the Decision Boundary, P, Considered in 10-Fold Cross-Validation
The y-axis shows the probability that PEP is recommended. The outcome is exclusively binary (0 or 1), but we jittered the points to avoid overlap and improve readability. This plot was made using the full survey dataset (none of the folds from the cross-validation were withheld).

eTable 2. Evaluation of the Decision Boundary for Logistic Regression Using 10-Fold Cross-Validation
We chose the decision boundary with the highest F-score (bold). F score was calculated as (2 * true positives) / (2 * true positives + false positives + false negatives), using the sums of each of these quantities across the 10 folds.

. Violin Plots of the Estimates From the Probability Model, US
There were 1,728 observations for Pr(rabid|exposure) and 41,472 observations for Pr(death|exposure). National estimates and pooled estimates according to the presence of terrestrial RABV variants were only included once (rather than repeated for each jurisdiction) and estimates from lick to intact skin (0 risk of rabies) were removed. Data sources for the estimates included the national rabies surveillance system (2011 -2020) and the literature.

eResults
Positivity rates. Overall positivity rates for rabies surveillance data collected between 2011 -2020 are shown in eTable 1. Positivity rates were highest for mongooses (0.83, 95% confidence interval [CI]: 0.78 -0.87) and skunks (0.29, 95% CI: 0.29 -0.30). The lowest positivity rate was among pocket pets (0, 95% CI: 0 -0.017). eFigure 2 shows the positivity rates for RABV across animal categories for each year from 2011 -2020. The positivity rates for most animals are consistent across this period, except for arctic fox and mongoose; however, these animals had an average of only 38 and 28 samples each year, respectively. Positivity rates for bats, cats, dogs, raccoons, skunks, and foxes are shown by U.S. jurisdiction in eFigures 3 -4. eFigure 5 shows the positivity rates for cattle, equine, hoofed animals, rodents of unusual size, and sheep and goats. The positivity rates were significantly lower for cattle, equine, and sheep and goats in jurisdictions without terrestrial RABV compared to those with terrestrial RABV. eFigure 6 shows positivity rates across bat species. The overall positivity rate for bats was 0. reported some probabilities for animals that were proven to be rabid, often by the death of one or more of the victims. 20, 23 Notably, diagnosis of RABV was not possible until 1903. 24 We categorized the estimates into four anatomical locations: head/neck, upper extremities, torso, and lower extremities. We noted in eTable 3 when a more specific location (e.g., hands or face) was reported.
The geographical location of the exposure is not relevant for these parameter estimates because the animals were either rabid or strongly suspected to be rabid. However, the species of animal is important because it is related to severity of exposure. Most exposures in our review were from rabid cats, dogs, or wolves and may not generalize well to rabies reservoir species in the United States; for example, multiple bites from a wolf would be more severe than multiple bites from a North American bat due to larger body size and teeth and hence, more inoculated virus.
Distributions for anatomical location and severity of exposure. We used distributions to summarize the data from eTable 3 to obtain eTable 4. We categorized the estimates into seven levels of severity: multiple bites, single bite, scratch or superficial bite (with teeth), scratch (with nails/claws), mucosal, lick to broken skin, and lick to intact skin. Ten out of 19 combinations of location and severity obtained from the literature were point estimates (eTable 3). We used a binomial distribution for one combination because the sample size was known. We used triangular distributions for the remaining combinations, following Cleaveland et al. (eTable 4). 25 A triangular distribution is a continuous probability distribution with lower limit a, upper limit b and mode c, where a < b and a ≤ c ≤ b. This distribution is often used when data are scarce. We identified the minimum and maximum values for the triangular distribution from the available data for each combination of location and severity. We used the midpoint of the minimum and maximum for the mode, c. For parameter estimates that had Shim et al. 22 as the only reference, we used the point estimate for the mode and the 95% confidence intervals as a and b. Alternatively, we could have used uniform distributions instead of triangular distributions.
eFigure 10 shows the probability distributions for single and multiple bite exposures. We only had one reference informing the distribution of single bite to the lower extremities. Based on the data for multiple bite exposures, we adjusted the parameter estimate for single bite to the lower extremities by subtracting 0.12 each from the minimum, mode, and maximum, so that the minimum and mode would be 0.
We used fold changes to estimate parameters for combinations of location and severity that were missing from the literature. We used the mode values for multiple bites and single bite to calculate fold changes (increased risk) for torso to upper extremities and for upper extremities to head/neck. The fold changes were used to impute missing values for other combinations of location and severity. Four combinations were missing: scratch or superficial bite (teeth) for both torso and upper extremities and scratch (nails/claws) for both head/neck and torso. The average fold change was 2.3 and 1.9 for torso to upper extremities and for upper extremities to head/neck, respectively, across multiple bites and single bite exposures. As we did not have enough information to calculate a fold change for lower extremities to torso, we used the same value for lower extremities as torso for scratch or superficial bite (teeth).
In addition to the missing combinations noted above, three parameter estimates for severity were not specific to a location: mucosal, lick to broken skin, and lick to intact skin. In each case, we applied the same non-specific estimate to all four location categories. The total number of location and severity combinations was 28 (4 locations times 7 levels of severity).

Risk model (updating probabilities).
We used the following equations to update the probabilities in our model: where Equation (1) is Bayes' rule, and Equation (2) is derived using the law of total probability.
The distributions of all the estimates from the probability model are shown in eFigure 9. We obtained 1,728 unique observations in the resulting dataset for Pr(rabid|exposure). The number of observations was determined in the following way: 2 jurisdiction-specific species can be vaccinated, so 2 species * 51 jurisdictions * 8 combinations of health status, exposure circumstances, and vaccination status = 816. The remaining 4 jurisdiction-specific species cannot be vaccinated, so 4 species * 51 jurisdictions * 4 combinations of health status and exposure circumstances. . The number of times PEP was recommended out of 24 scenarios (sum of "agree" and "strongly agree") was 18.2 on average (range 11 -24). There was a non-linear (triangular) correlation between years of experience and both the number of times PEP was recommended and risk profile (eFigure 12). This relationship suggests that there is greater variability in PEP recommendations among public health officials with less experience compared to those with more experience. Risk profile was calculated by summing: Strongly agree = 1, Agree = 2, Disagree = 3, Strongly disagree = 4. The higher the number, "riskier" the respondent is.
Determining the decision boundary for p in logistic regression. eTable 2 shows the performance of the logistic regression model from 10-fold cross-validation for each value of the decision boundary, p, considered. We selected p = 0.5 for having the highest F-score. This decision boundary was associated with a risk threshold of approximately 0.0004 in the full dataset (including all the folds). eFigure 8 shows the risk thresholds associated with each p. There are several false positives using our decision boundary of 0.5. These points are in the lower right corner of eFigure 8. False positives are not as concerning as false negatives because it means our recommendations are conservative compared to those of the survey respondents. In other words, the tool would recommend PEP in situations when the subject matter experts do not recommend PEP. False positives may also reflect biases in the surveillance data, which we do not necessarily want to correct for at the expense of incorrectly classifying high-risk exposures as PEP = no. Based on rabies risk assessment guidelines, we expect the surveillance data are over-representative of high-risk exposures. They could also reflect experts making recommendations that are inconsistent with data.
Model diagnostics for logistic regression. We obtained a Pseudo-R² (McFadden) of 0.16, suggesting moderate fit. Pseudo R 2 ranges between 0 and 1; however, values near 1 are exceedingly rare for real data. Values between 0.2 and 0.4 are considered a very good fit. 26 We checked the four main assumptions of logistic regression, (1) the outcome is binary, (2) the relationship between the logit of the outcome and each predictor variable is linear, (3) there are no extreme outliers in the predictors, and (4) there is no multicollinearity among predictor variables. (4) is not relevant, as we only have one predictor. The outcome is binary, as we collapsed the survey responses into PEP yes vs no. eFigure 13 shows the plot for (2), demonstrating that the assumption of linearity is met. We identified 73 potentially influential observations in our data that had Cook's distance > 4/N, where N is the total number of data points, 1,194 (eFigure 13); however, all standardized residuals had an absolute value < 3. Among the 73 potential outliers, there were 30 instances where PEP was recommended, yet Pr(rabid|exposure) was low, and there were 43 instances where PEP was not recommended but Pr(rabid|exposure) was high (0.02 >). Omitting these observations, we obtained a risk threshold of 0.0008.
Sensitivity analyses on survey data. When we only included respondents with at least 9 years of experience, we found a similar risk threshold compared to the main analysis: 0.0003.
Using only survey responses consisting of "strongly agree" and "strongly disagree," we obtained a risk threshold of 0.0004 which was the same as that obtained in the main analysis.
Outliers in survey data. Out of 1,194 non-missing observations (survey scenario matched with underlying Pr(rabid|exposure) estimate), 194 (16%) observations had a median above the risk threshold, yet PEP was not recommended. Of these, 167 were "disagree" and only 27 were "strongly disagree." For "strongly disagree" (eTable 5), 2 out of 27 observations involved animal categories for which a national estimate was used.